Method for transmitting stream between electronic devices and electronic device for the method thereof

ABSTRACT

A technique for transmitting a video stream between a first electronic device and a second electronic device is provided. The first device receives an encoded video stream and determines whether an encoding format of the encoded video stream is a format decodable by a second electronic device. If the format is decodable, the encoded video stream is transmitted without a re-encoding operation to the second device. The video stream is displayed at the first device delayed by a determined delay time enabling the video stream to be displayed at the second device substantially synchronized with the display at the first device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.13/768,402 filed on Feb. 15, 2013 which claims the benefit under 35U.S.C. §119(a) to a Korean patent application filed in the KoreanIntellectual Property Office on Feb. 24, 2012, and assigned Serial No.10-2012-0018922, the entire disclosure of which is hereby incorporatedby reference.

TECHNICAL FIELD

The present disclosure relates generally to a method and electronicdevice for transmitting a video stream, and more particularly, fortransmitting a video stream between electronic devices that communicatewith each other according to a short range communication scheme.

BACKGROUND

Communication techniques between electronic devices are advancing invarious ways. In particular, with advances in communication technologiesfor portable terminals and introduction of Internet connectivity,communication between electronic devices including portable terminals isutilized for many purposes.

Wi-Fi communication is a Wireless Local Area Network (WLAN) technology.WLAN provides a communication service such as Internet to portableterminals (e.g., smart phones, tablet PCs, Personal Digital Assistants(PDAs), notebook computers, etc.) traveling within a certain distancefrom an installed Access Point (AP) which is a wireless access device.WLAN technology had a short propagation distance of about 10 m in theearly stage but has recently greatly expanded the propagation distanceup to 50 m˜hundreds of meters. Data transfer rates in WLAN have alsoimproved to transmit and receive high-volume multimedia data.

The advance of Wi-Fi communication has enabled the development of aWi-Fi Direct technique which allows print and content sharing throughfree communication between Wi-Fi devices without an AP or a router.Wi-Fi Direct, which is also referred to as Wi-Fi Peer-to-Peer (P2P), canoperate by grouping a P2P group owner and a P2P client.

A protocol emerging in this field, called Wi-Fi Display communication,is a technique for sharing data including a video stream and/or an audiostream between Wi-Fi enabled electronic devices. For example, a Wi-FiDisplay source electronic device (hereafter, “source device” or just“source”) transmits a video stream and/or an audio stream to a Wi-FiDisplay sink electronic device (hereafter, “sink device” or just“sink”). The two devices share the streams and simultaneously output thestreams through a screen and/or a speaker. In Wi-Fi Displaycommunication, connectivity is achieved using, for example, Wi-FiDirect.

Herein, a source device denotes a device transmitting the video streamand/or the audio stream, and a sink device denotes a device receivingthe video stream and/or the audio stream.

A standard for the Wi-Fi Display communication is described in a Wi-FiDisplay specification defined by the Wi-Fi alliance, which is aconsortium of companies and entities establishing wireless protocols.

The Wi-Fi Display communication for transmitting data from the Wi-FiDisplay source device to the Wi-Fi Display sink device includes Wi-FiDisplay (WFD) device discovery, WFD connection setup, WFD capabilitynegotiation, and WFD session establishment. Besides these, the WFDcommunication can further include a plurality of selective operationssuch as link content protection.

When the WFD session is established, data is transmitted from the WFDsource device to the WFD sink device.

WFD session establishment, and the stream transmission from the WFDsource device to the WFD sink device, will now be explained. The WFDsource device decodes an encoded stream which is stored therein orreceived from another medium such as Internet, and outputs the decodedvideo stream through the screen and the audio stream through thespeaker. Herein, encoding refers to at least compressing of the videostream and/or the audio stream. For example, the video stream iscompressed in the format of Moving Picture Expert Group (MPEG)-4 ofInternational Standard Organization (ISO)-International ElectronicCommission (IEC) or H264 of International TelecommunicationsUnion-Telecommunication standard sector (ITU-T), and the audio stream iscompressed in the format of Linear Pulse Coded Modulation (LPCM) 44.1,LPCM 16-48, Advanced Audio Coding (AAC), AC-3, Authentication,Authorization, and Accounting (AAA), Adaptive Multi Rate (AMR), WindowsMedia Audio (WMA), MPEG-2 Audio, or MPEG-4 Audio.

The WFD source device encodes the raw streams (e.g., video stream andaudio stream) that have been decoded and output through thescreen/speaker, and transmits the newly encoded stream to the WFD sinkdevice. The WFD sink device decodes the encoded stream received from theWFD source device, and outputs the video stream through the screen andthe audio stream through the speaker. Hence, the WFD source device andthe WFD sink device share the audio and video streams.

Thus, the WFD source device decodes the encoded stream stored therein orreceived through another medium such as Internet, outputs theaudio/video streams through the screen/speaker, and re-encodes thestreams in order to transmit the output raw stream. However, encodingthe video stream is time consuming. The video stream requires anencoding time about 10 times that of the audio stream, which may differaccording to performance of a hardware encoder or a software encodingmodule. In a camera application, a portable communication terminal,which is an example of the WFD source device, often captures an imageusing an embedded digital camera when taking a video, a large number ofimages are captured using continuous shooting. With the large amount ofdata generated, encoding is necessary to store or send the capturedimages to another person. As the portable terminal frequently uses theencoder (or the software encoding module) for encoding images and theamount of the image data to process increases, this puts a strain ondevice resources.

Hence, in the WFD communication, it is required to address the muchencoding time of the WFD source electronic device by modifying themethod of the WFD source electronic device for decoding the encodedvideo stream, outputting the raw video stream through the screen, andre-encoding and transmitting the output video stream.

BRIEF SUMMARY OF THE INVENTION

To address the above-discussed deficiencies of the prior art, it is aprimary aspect of the present invention to provide an electronic devicefor improving a method for encoding and transmitting a decoded outputstream.

Another aspect of the present invention is to provide a streamtransmitting method for improving encoding and transmission of a decodedoutput stream.

Yet another aspect of the present invention is to provide an electronicdevice for improving a method for encoding and transmitting a decodedoutput video stream.

Still another aspect of the present invention is to provide atransmitting method for improving encoding and transmission of a decodedoutput video stream.

A further aspect of the present invention is to provide a video streamreceiving device which corresponds to the improved electronic device.

A further aspect of the present invention is to provide a video streamreceiving method which corresponds to the improved transmitting method.

According to an exemplary embodiment of the present invention, a methodoperable in a first (source) electronic device includes determiningwhether an encoding format of an encoded video stream is a formatdecodable by a second electronic device, and decoding and outputting aframe of the encoded video stream through an output device of the firstelectronic device, and transmitting the encoded video stream to thesecond electronic device wherein the transmitting the encoded videostream to the second electronic device is transmitting the encoded videostream without re-encoding it after delaying by a determined time fromframe time information of the encoded video stream.

According to an exemplary embodiment of the present invention, a methodfor processing a video stream in a second electronic device includesconfirming a delay time for delaying and outputting an encoded videostream, from a frame of the encoded video stream received from a firstelectronic device; and decoding and outputting the encoded video streamthrough an output device of the second electronic device, by delayingthe video stream by the delay time.

According to an exemplary embodiment of the present invention, first andsecond electronic devices each include one or more processors; a memory;and one or more modules stored to the memory to be executed by the oneor more processors. The module includes instructions to carry outmethods in accordance with the invention.

Computer readable storage media store one or more programs whichcomprise instructions which, when executed by an electronic device,cause the device to execute methods in accordance with the invention.

Other aspects, advantages, and salient features of the invention willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses exemplary embodiments of the invention.

Certain embodiments of the invention may exhibit the followingadvantage: when the stream is transmitted from the source device to thesink device, it is possible to enhance the method of the source devicefor decoding the encoded video stream, outputting the raw video streamthrough the screen, and re-encoding and transmitting the output videostream.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainexemplary embodiments of the present invention will be more apparentfrom the following description taken in conjunction with theaccompanying drawings, in which:

FIG. 1A, FIG. 1B and FIG. 1C are simplified block diagrams illustratingtransmission of signals between source and sink devices in accordancewith embodiments of the invention;

FIG. 2 is a block diagram of a WFD source device according to anexemplary embodiment of the present invention;

FIG. 3 is a block diagram of a processor unit of FIG. 2 according to anexemplary embodiment of the present invention;

FIG. 4 is a flowchart of WFD communication session establishment fortransmitting the stream from the WFD source device to the WFD sinkdevice according to an exemplary embodiment of the present invention;

FIG. 5A is a flowchart of WFD communication session establishment fortransmitting the video stream from the source device to the sink deviceaccording to an exemplary embodiment of the present invention;

FIG. 5B is a diagram of an apparatus, which corresponds to the method ofthe flowchart of FIG. 5A, for establishing the WFD communication sessionfor transmitting the video stream from the source device to the sinkdevice according to an exemplary embodiment of the present invention;

FIG. 6A is a flowchart of WFD communication session establishment fortransmitting the video stream from the source device to the sink deviceaccording to another exemplary embodiment of the present invention;

FIG. 6B is a diagram of an apparatus, which corresponds to the method ofthe flowchart of FIG. 6A, for establishing the WFD communication sessionfor transmitting the video stream from the source device to the sinkdevice according to another exemplary embodiment of the presentinvention;

FIG. 7A is a flowchart of the sink electronic device, which receives thevideo stream from the source device, for receiving and outputting thevideo stream according to another exemplary embodiment of the presentinvention;

FIG. 7B is a diagram of an apparatus, corresponding to the flowchart ofFIG. 7A, for receiving and outputting the video stream of the sinkdevice which receives the video stream from the source device accordingto another exemplary embodiment of the present invention;

FIG. 8 is a flowchart of WFD communication session establishment fortransmitting the video stream from the source device to the sink deviceaccording to another exemplary embodiment of the present invention;

FIG. 9 is a timing diagram illustrating exemplary timing for the variousoperations that occur in the method of FIG. 5A; and

FIG. 10 is a timing diagram illustrating exemplary timing for variousoperations that occur in the method of FIG. 6A.

Throughout the drawings, like reference numerals will be understood torefer to like parts, components and structures.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of exemplaryembodiments of the invention as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the embodiments described hereincan be made without departing from the scope and spirit of theinvention. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of theinvention. Accordingly, it should be apparent to those skilled in theart that the following description of exemplary embodiments of thepresent invention is provided for illustration purpose only and not forthe purpose of limiting the invention as defined by the appended claimsand their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

By the term “substantially” it is meant that the recited characteristic,parameter, or value need not be achieved exactly, but that deviations orvariations, including for example, tolerances, measurement error,measurement accuracy limitations and other factors known to those ofskill in the art, may occur in amounts that do not preclude the effectthe characteristic was intended to provide.

The present invention relates to a method for transmitting at least avideo stream between electronic devices which communicate with eachother according to a short range communication scheme, and an electronicdevice for performing the method. Examples of a suitable short rangecommunication scheme include, but are not limited to, a Wi-Fi Displayscheme, a Bluetooth scheme, a short range wire communication scheme, anear field communication scheme and wireless local area network scheme.Hereunder, embodiments of the invention will be described in the contextof a Wi-Fi Display scheme, but it is understood that other communicationenvironments are also within the scope of the invention.

FIGS. 1A to 1C are simplified block diagrams illustrating transmissionof signals between source and sink devices in accordance withembodiments of the invention. Referring to FIG. 1A, Wi-Fi Display (WFD)compatible source device 110 (hereafter referred to as “the sourcedevice” or just “the source” for brevity) transmits audio and videosignals to a WFD compatible sink device 120 (hereafter referred to as“the sink device” or just “the sink” for brevity). An audio/video (A/V)signal that is stored on a first media device and transmitted to andplayed on another media device is said to be “streamed” to the seconddevice; thus, “audio signal” and “video signal” as used herein will beinterchangeably referred to as an “audio stream” and a “video stream,”respectively. An audio or video “stream” herein will also refer to anaudio/video frame sequence read from memory within the same device. Theaudio and video signals are also referred to as audio and video payloadsin Wi-Fi Display specifications. In the various embodiments herein, thesource device may have pre-stored the A/V signals that are subsequentlystreamed to the sink device. Or, just prior to streaming the A/V signalsto the sink, the A/V signals are received by the source from an externaldevice such as from the Internet, e.g., in real time, and momentarilystored. Control signals are exchanged between the source and sinkdevices when necessary. For example, a control signal can be sent fromthe source to the sink so as to control the sink to play, stop, or pausethe video and/or audio stream.

The WFD-based communication for transmitting data from the source to thesink includes WFD device discovery, WFD connection setup, WFD capabilitynegotiation, and WFD session establishment. Further, the WFDcommunication selectively executes link content protection.

In the WFD communication of FIG. 1A, when the source device transmitsthe video stream to the sink device, the source transmits its storedencoded video stream to the sink, rather than re-encoding the decodedvideo stream output through an output device (a screen) of the sourceand streaming the re-encoded video to the sink. In so doing, the sourcedevice delays the encoded video stream by a certain time from frame timeinformation of the encoded video stream (described further below inconnection with FIGS. 9-10). In the WFD communication of FIG. 1A, whenthe source device transmits the video and audio streams to the sinkdevice, the video stream and audio stream transmitted can bedifferentially processed. Here, such differential processing transmitsthe encoded video stream stored at the source device, to the sinkdevice. However, the differential processing re-encodes the decodedaudio stream output through the speaker of the source device, andtransmits the re-encoded audio stream. That is, for the video stream,the source transmits the encoded video stream stored therein, to thesink or transmits the video stream including output delay information,rather than decoding the encoded video stream, re-encoding the raw videostream output through the screen of the source, and transmitting are-encoded video stream to the sink. Alternatively, no differentialprocessing is used, such that the audio stream can also be handled in ananalogous manner as the video stream, i.e., it is transmitted to thesink device in its original encoded form, without decoding andre-encoding.

In the operations of FIGS. 1A, 1B, and 1C, signals are encoded, whichincludes a compression method, to reduce the amount of data needed forstorage and transmission. The encoding converts the type or form of theinformation into other types or forms for standardization, security,processing rate enhancement, or storage space saving. For example, thevideo stream and/or the audio stream is/are compressed according to acertain format. Suitable examples for video stream compression includeMPEG-4 or H264 format, examples for the audio stream are LPCM44.1,LPCM16-48, AAC, AC-3, AAA, AMR, WMA, MPEG-2 Audio or MPEG-4 Audioformat.

The decoding converts the encoded information back to the originalinformation. Mostly, the decoding is the reverse process of theencoding. The decoding decompresses the encoded audio and/or videostreams and outputs the raw stream. The video raw stream indicates, forexample, Luminance, Blue-Y, Red-Y (YUV) or Red, Green, Blue (RGB)stream.

FIG. 1B depicts an example in which only a video stream is transmittedfrom a WFD source device 130 to a WFD sink device 140 in the WFDcommunication according to an exemplary embodiment of the presentinvention. Here, the control signal function may be substantially thesame as for that of FIG. 1A. The WFD communication session establishmentcan also be the same as in FIG. 1A.

In the WFD communication of FIG. 1B, the video can be streamed from thesource to the sink in the same manner as explained above for FIG. 1A.That is, when the source transmits the video stream to the sink device,the source transmits its stored encoded video stream to the sink, ratherthan re-encoding the decoded video stream output through its outputdevice (the screen) and transmitting the re-encoded video stream to thesink. In so doing, the source delays the encoded video stream by acertain time from the frame time information of the encoded videostream, or may include the output delay information to the video stream.

FIG. 1C depicts an example in which the video stream is transmitted froma WFD source device 150 to a primary WFD sink device (hereafter,referred to as a first sink device) 160 and the audio stream istransmitted to a secondary WFD sink device (hereafter, referred to as asecond sink device) 170 in the WFD communication according to anexemplary embodiment of the present invention. Here, the control signalfunction has been explained in FIG. 1A. The session establishment in theWFD communication as described in FIG. 1A is applied between the sourceand the first sink device and between the source and the second sinkdevice in FIG. 1C. In this example, the first sink device and the secondsink device are coupled. For example, the coupling allows the primarysink 160 and secondary sink 170 to share the audio and video streamsthat are streamed to them by the source device 150.

In the WFD communication of FIG. 1C, when the source transmits the videostream to the first sink, the source transmits its stored encoded videostream to the first sink, rather than re-encoding the decoded videostream output through its output device (the screen) and transmittingthe re-encoded video stream to the sink. In so doing, the source delaysthe encoded video stream by a certain time from the frame timeinformation of the encoded video stream, and may transmit the videostream including the output delay information.

When the video stream and the audio stream are transmitted to separatesink devices, as in the case of FIG. 1C, the video stream and the audiostream can be processed and transmitted in different fashions. Inparticular, the video streamed to the first sink 160 can be theoriginally encoded video, i.e., without re-encoding it, whilst the audiostreamed to the second sink 170 can be re-encoded audio, as describedabove for FIG. 1A. FIG. 2 is a block diagram of a WFD device, 200, whichcan operate as either any source or sink device to perform theoperations described in connection with FIGS. 1A-1C. The followingdescription will pertain to its operation as a source device. Sourcedevice 200 can be a portable (i.e., handheld) electronic device or aportable terminal, e.g., smart phone, mobile phone, mobile pad, mediaplayer, tablet computer, handheld computer, digital camera, or PersonalDigital Assistant (PDA). The source device 200 may be a portableelectronic device combining two or more functions of those devices.

The source device 200 includes a memory 210, a processor unit 220, afirst wireless communication subsystem 230, a second wirelesscommunication subsystem 231, an external port 260, an audio subsystem250, a speaker 251, a microphone 252, an Input/Output (I/O) system 270,a touch screen 280, and other input or control devices 290. A pluralityof memories 210 and a plurality of external ports 260 can be used.

The processor unit 220 can include a memory interface 221, processor222, and a peripheral interface 223. Memory interface 221, processor222, and/or the peripheral interface 223 can be separately provided orintegrated onto one or more integrated circuits.

The processor 222 (which may actually comprise multiple processors)performs various functions for the electronic device 200 by runningvarious software programs, and processes and controls voicecommunication and data communication. In addition to such typicalfunctions, the processor 222 also executes software modules (instructionsets) stored in memory 210 and performs various particular functionscorresponding to the modules. That is, the processor 222 carries outmethods according to exemplary embodiments of the present invention inassociation with software modules stored in memory 210.

Hereafter, an encoded video stream received by a source device from anexternal source such as the Internet, or retrieved from memory of thesource device for reproduction and display on the device's screen willbe referred to simply as “the encoded video stream”. When the videostream is received from an external source, it is temporarily stored inmemory prior to decoding. In various embodiments, the processor 222compares an encoding format of the encoded video stream to an encodingformat(s) supported by a target sink device. When the encoding format ofthe encoded video stream is supported for decoding in the sink device,the encoded video stream is streamed to the sink device without are-encoding operation. The processor 222 decodes the encoded videostream by executing a CODEC module 215 of the memory 210, and outputsthe decoded video stream through a touch screen 280. In order to displaythe reproduced video streams at the source and transmit the streams tothe sink devices synchronously, the encoded video stream is delayed by acertain time from the frame time information of the encoded video streamand then transmitted to the sink electronic device. Alternatively, inorder to inform the sink device that transmitted encoded video stream isnon-decoded video stream and therefore it needs time to decode theencoded video streams at the source device, the encoding video streamincluding the output delay time information is transmitted to the sinkelectronic device. (Both of these approaches are explained below inconnection with FIGS. 9-10.)

The processor 222 can include one or more data processors, an imageprocessor, and a CODEC (realized by executing the instructions of CODECmodule 215). The data processor, the image processor, and the CODEC maybe separately provided. Alternatively, processor 222 includes aplurality of processors for performing different functions. Theperipheral interface 223 connects the I/O subsystem 270 of theelectronic device 200 and various peripherals to the processor 222. Amemory interface 221 connects processor 222 to the memory 210.

The various components of the source electronic device 200 can becoupled using one or more communication buses or one or more streamlines.

The external port 260 is used to connect the electronic device 200 toother electronic devices directly or indirectly via a network (e.g.,Internet, intranet, and wireless LAN). The external port 260 can be, forexample, but not limited to, Universal Serial Bus (USB) port or aFIREWIRE port.

A motion sensor 291 and a first optical sensor 292 are coupled to theperipheral interface 223 to allow various functions. For example, themotion sensor 291 and the first optical sensor 292 are coupled to theperipheral interface 223 to detect motion of the electronic devices andthe light from the outside. Besides these, a positioning system andother sensors such as temperature sensor or bionic sensor can be coupledto the peripheral interface 223 to perform their functions.

A camera subsystem 293 is coupled to a second optical sensor 294 toperform camera functions such as photo and video clip recording.

The first optical sensor 292 and the second optical sensor 294 canemploy a Charged Coupled Device (CCD) or a Complementary Metal-OxideSemiconductor (CMOS) device.

The communication function is conducted through the one or more wirelesscommunication subsystems 230 and 231. The wireless communicationsubsystems 230 and 231 can include radio frequency receiver andtransmitter and/or optical (e.g., infrared light) receiver andtransmitter. The first wireless communication subsystem 230 and thesecond wireless communication subsystem 231 can be distinguished basedon the communication network of the electronic device 200. For example,the communication network can include a communication subsystem designedto operate over, but not limited to, a Global System for Mobilecommunication (GSM) network, an Enhanced Data GSM Environment (EDGE)network, a Code Division Multiple Access (CDMA) network, a W-CDMAnetwork, a Long Term Evolution (LTE) network, an Orthogonal FrequencyDivision Multiple Access (OFDM) network, a Wi-Fi network, a WiMaxnetwork and/or a Bluetooth network. For WFD implementations of theinvention, for instance, Wi-Fi communication is performed over a Wi-Finetwork using the first subsystem 230. In this case, the secondsubsystem 231 can be designed for short-range communication that enablesdata streaming to a sink device. The first wireless communicationsubsystem 230 and the second wireless communication subsystem 231 may beintegrated into a single wireless communication subsystem.

The audio subsystem 250 can be coupled to the speaker 251 and themicrophone 252 to process the input and the output of the audio streamsuch as voice recognition, voice reproduction, digital recording, andtelephone function. That is, the audio subsystem 250 communicates with auser through the speaker 251 and the microphone 252. The audio subsystem250 receives the data stream through the peripheral interface 223 of theprocessor unit 220 and converts the received data stream to an electricsignal. The converted electric signal is fed to the speaker 251. Thespeaker 251 converts the electric signal to a sound wave audible by theuser and outputs the sound wave. The microphone 252 converts the soundwave from the user or other sound sources to an electric signal. Theaudio subsystem 250 receives the converted electric signal from themicrophone 252. The audio subsystem 250 converts the received electricsignal to the audio data stream and sends the converted audio datastream to the peripheral interface 223. The audio subsystem 250 caninclude an attachable and detachable ear phone, head phone, or head set.

The I/O subsystem 270 can include a touch screen controller 271 and/oran other input controller 272. The touch screen controller 271 can becoupled to the touch screen 280. The touch screen 280 and the touchscreen controller 271 can detect the contact and the motion fordetecting touch gestures using, but not limited to, capacitive,resistive, infrared and surface sound wave techniques for determiningone or more contact points with the touch screen 280 and a multi-touchdetection technique including various proximity sensor arrays or otherelements. The other input controller 272 can be coupled to the otherinput/control devices 290. The other input/control devices 290 caninclude one or more up/down buttons for controlling the volume of thespeaker 251 and the microphone 252. The button can employ a push buttonor a rocker button. The other input/control devices 290 can employ arocker switch, a thumb wheel, a dial, a stick, and/or a pointer such asstylus.

The touch screen 280 provides the I/O interface between the electronicdevice 200 and the user. That is, the touch screen 280 forwards theuser's touch input to the processor 220. The touch screen 280 alsoincludes a display that functions as a medium for displaying the outputof the electronic device 200 to the user. Visual output on the displaycan be represented as text, graphic, video, and a combination of these.Thus the encoded video stream is decoded and the resulting video isreproduced on the display of the touch screen.

The touch screen 280 can employ various displays, examples of whichinclude but are not limited to, Liquid Crystal Display (LCD), LightEmitting Diode (LED), Light emitting Polymer Display (LPD), Organic LED(OLED), Active Matrix OLED (AMOLED) or Flexible LED (FLED).

The memory 210 can be coupled to the memory interface 221. The memory210 can include fast random access memory (RAM) such as one or moremagnetic disc storage devices and/or non-volatile memory, one or moreoptical storage devices, and/or a flash memory (e.g., NAND and NOR).When the encoded video stream is received by device 200 from an externalsource, it is temporarily stored in the RAM during the course of thedecoding process.

The memory 210 stores software. Software components include an operatingsystem module 211, a communication module 212, a graphic module 213, auser interface module 214, a CODEC module 215, a camera module 216, andone or more application modules 217. The modules being the softwarecomponents can be represented as a set of instructions, and thus themodule can be referred to as an instruction set. Also, the module may bereferred to as a program.

One or more modules including the instructions for performing thevarious methods of the present invention can be stored to the memory210.

The operating system software 211 (the embedded operating system such asWINDOWS, LINUX, Darwin, RTXC, UNIX, OS X, or VxWorks) includes varioussoftware components for controlling general system operations. Theseinclude, e.g., memory management and control, storage hardware (device)control and management, and power control and management. The operatingsystem software 211 processes the normal communication between varioushardware (devices) and software components (modules).

The communication module 212 allows communication with an externalelectronic device such as a computer, server, and/or portable terminal,through the wireless communication subsystems 230 and 231 or theexternal port 260.

The graphic module 213 includes various software components forproviding and displaying graphics on the touch screen 280. The term‘graphics’ encompasses text, webpage, icon, digital image, video, andanimation.

The user interface module 214 includes various software componentsrelating to a user interface. The user interface module 214 is involvedin the status change of the user interface and the condition of the userinterface status change.

The CODEC module 215 can include software components relating to theencoding and the decoding of the video file. The CODEC module 215 caninclude a video stream module such as MPEG module and/or H204 module.The CODEC module 215 can include various audio file CODEC modules forAAA, AMR, and WMA. The application module 217 may include instructionsets for carrying out methods of the present invention as describedherein. Frame time information of the encoded video stream stored to thesource electronic device can obtain time information of the whole frameand/or time information per frame in a packet header (a frame header) ofthe parsed encoded video stream or a packet payload (frame payload)region. For example, the time information is contained in dataindicating how the audio stream (data) and the video stream (data) areinterleaved in the parsed encoded stream. The corresponding frame-basedtime information can include the output order of the correspondingframe. The corresponding frame-based time information can be a PresentTime Stamp (PTS) value being information relating to when to display theframe. The frame includes a Program Clock Reference (PCR) correspondingto the clock, to thus check the time.

To output (i.e., display) the encoded video stream, the frame timeinformation PTS of the decoded video stream can be acquired bymonitoring the output frame of the decoded video stream.

Thus, for example, a difference value can be determined by comparing theframe time information PTS of the encoded video stream and the frametime information PTS of the decoded video stream output.

The camera module 216 includes camera related software componentsallowing camera related processes and functions. The application module217 may further include a browser, an e-mail, an instant message, a wordprocessing, keyboard emulation, an address book, a touch list, a widget,Digital Right Management (DRM), voice recognition, voice reproduction, aposition determining function, and a location based service. The memory210 can include an additional module (instructions) in addition to thosemodules. Alternatively, if necessary, part of the modules (instructions)may not be used.

The various functions of the electronic device 200 as stated above andto be explained, can be executed by hardware and/or software and/ortheir combination including one or more stream processing and/orApplication Specific Integrated Circuits (ASICs).

FIG. 3 is a block diagram depicting an alternative embodiment of aprocessor unit, 300, that may be used in place of processor unit 220 ofFIG. 2. Processor unit 300 includes a memory interface 310, a peripheralinterface 340, a processor 320, and a CODEC 330. Processing unit 300differs from processing unit 220 by implementing the CODEC 330 inhardware, rather than the software implementation of the CODEC functionas in processing unit 220. The processor 320 is configured to executemethods according to the present invention in association with the CODEC330 and via the execution of the software modules stored to the memory210. That is, processor 320 and the CODEC 330 can together carry outmethods according to the present invention as exemplified herein.

When the device 200 is used as a sink device, i.e., used in a sinkdevice mode, the processor 220 controls corresponding operations tocarry out the sink device functions described herein, in accordance withinstructions read from memory 210. Such operations can include receivingand processing the video stream and the audio stream from the sourcedevice, decoding the video stream and displaying reproduced video insync with the video reproduction in the source device. Decoding iscarried out in the same fashion as for the source device mode.

FIG. 4 is a flowchart of WFD communication session establishment fortransmitting the AN stream from the WFD source device to the WFD sinkdevice according to an exemplary embodiment of the present invention. Tocommence the WFD communication, WFD device discovery 410, WFD connectionsetup 420, WFD capability negotiation 430, and WFD session establishment440 are performed. For secure data transfer, link content protection(not shown) is also selectively conducted. After the WFD session isestablished, the video stream and the audio stream can be transmittedbetween the WFD devices (from the source to the sink device).

In the WFD device discovery 410, the source and sink devices confirm themutual presence. The WFD connection setup 420 determines which methoduses the WFD connection. For example, the WFD connection setupdetermines Wi-Fi or other short range communication parameters to usefor the WFD connection. The WFD capability negotiation 430 determinesthe sequence of the message exchange between a Real Time StreamingProtocol (RTSP) source device and a RTSP sink device. The RSTP messageexchange sequence is determined in order to determine parametersdefining the video/audio streams transmitted from the source device tothe sink device. In this step, the encoding format (i.e., encodingformat information) of the video stream decodable by the sink device canbe transmitted from the sink to the source device. Examples of the videostream format information include but are not limited to H264 or MPEG-4.When determining a sequence of message exchange between the source andsink devices and determining parameters of the transmitted video/audiostreams, the sink device can transmit the encoding format information ofthe decodable video stream to the source device.

The WFD session establishment 440 establishes the session between thesource and sink devices. When the session is established, the sourcedevice can transmit its selected stream to the sink device. The streamis transmitted between the source device and the sink device accordingto the present method. In addition, a plurality of optional steps suchas link content protection for security of the content transmitted fromthe source to sink devices can be conducted in necessary phases. Thelink content protection can configure a High-bandwidth Digital ContentProtection (MCP) 2.0 session key in the content.

FIG. 5A is a flowchart of the WFD communication session establishmentfor transmitting the video stream from the source device to the sinkdevice according to an exemplary embodiment of the present invention.The source device checks the encoding format of the encoded video streamand compares the encoding format to a format decodable and displayableby the sink device in step 510. To this end, prior to the comparison,the source device obtains information of at least one decodable formatof the sink device. That is, before checking the encoding format of theencoded video stream, the source device can receive the encoding format(i.e., encoding format information) of video decodable by the sinkdevice, from the sink device Examples of encoding formats include butare not limited to H264 and MPEG-4. The sink device may be capable ofdecoding a plurality of encoding formats of the video stream. Theencoding format information of video decodable by the sink device can bereceived in, for example, the WFD capability negotiation 430 of FIG. 4.That is, when determining the sequence of message exchange between thesource device and the sink device and determining the parameters of thetransmitted video/audio streams, the encoding format information ofdecodable video can be received from the sink device. The sink devicecan transmit the encoding format information of the video stream using acontrol signal, a frame header (e.g., a packet header in a packettransmission), or a particular location of the frame payload (e.g., apacket payload of a packet transmission).

Before checking the encoding format of the encoded video stream in step510, the source device can send a request to the sink device the requestinquiring about whether the encoding format of the encoded video stream,is a decodable format of the sink device, and can receive a response tothe request from the sink device, which are not depicted in the drawing.These operations can be performed in the WFD capability negotiation 430of FIG. 4.

In step 520, the source device decodes the video stream and outputs thevideo on its own display. In addition, if determined in the formatcomparison of step 510 that the sink device's decoding format iscompatible with the format of the encoded video stream, the encodedvideo stream is transmitted to the sink device. This is done without adecoding and re-encoding operation as in conventional methods. If theencoded video stream is part of an interleaved A/V stream (“the A/Vstream”) with video data interleaved with audio data, the video streamis first extracted from the A/V stream before decoding and transmitting.That is, the A/V stream can be parsed and split to the encoded audiostream and the encoding video stream. The A/V stream can be, but notlimited to, an encoded data stream downloaded from another electronicdevice and stored to the source device, or obtained from anothercomponent such as a camera subsystem of the source device, or receivedas a data stream streamed over the Internet. The parsing can be realizedusing hardware or software. Using software, an instruction set for theparsing can be provided as a separate parsing module or as part of theCODEC module and stored to the memory 210 of FIG. 2. The parsed videoand audio streams are stored to the source device. The storagelocation(s) can be part of the memory interface, part of a peripheralinterface, or a separate buffer memory (RAM).

When the decoded video stream is displayed at the source device and theencoded video stream is transmitted in step 520, intentional delays canbe introduced at either one or both of the source and sink devices, inorder that the display of the encoded video stream at the source and thetransmission of the encoded video stream to the sink devices are insynchronization. Frame time information of the encoded video stream canbe obtained from the time information of the whole packet and/or thetime information per frame constructing the packet, in the packet header(frame header) or the packet payload (frame payload) of the parsedencoded video stream. The frame time information can correspond toPicture Parameter Set (PPS) and Sequence Parameter Set (SPS) in the MPEGformat. The corresponding frame time information can include the outputorder of the corresponding frames. The corresponding frame timeinformation can be the PTS value which is information about when thecorresponding frame is displayed. Since the frame includes the PCRcorresponding to the clock, the time can be checked.

The frame time information PTS of the decoded video stream to output theencoded video stream can be acquired by monitoring the output frame ofthe decoded video stream.

Hence, for example, the time taken for decoding can be calculated as atime difference value determined by comparing the frame time informationPTS of the encoded video stream and the frame time information PTS ofthe decoded and output video stream.

In addition, a link delay for transmitting the video stream from thesource to the sink can be determined. The link delay includes modulationand other processing delays occurring at the source device, andpropagation delay. In this case, the time difference may be the timetaken for decoding at the source device extracted by the link delay.

Further, a receiver/demodulation delay at the sink device (“sink delay”)can be determined. The sink delay includes demodulation delays at thesink device. In this case, the time difference may be the time taken fordecoding at the source device extracted both by the link delay and thesink delay. The “link delay” may be determined as ½ of a Round Trip Time(RTT) value between the source device and the sink device. The RTTindicates the time taken for the source device to receiveacknowledgement from the sink device in response to the transmittedstream. A method for measuring the RTT value can adopt any suitable oneof well-known methods.

In addition, a sink device decoding time, i.e., the time taken for thesink device to receive, decode, and output the encoded video streamthrough its output device can be measured by the sink device. That is,the received frame time information of the encoded video stream can beconfirmed in the packet header (frame header) or the packet payload(frame payload) of the parsed encoded video stream. The frame timeinformation can correspond to the PPS and the SPS in the MPEG format. Asthe frame includes the PCR corresponding to the clock, the time can bechecked. The output time of the video stream decoded to output theencoded video stream can be obtained by monitoring the output frame ofthe decoded video stream.

The time value from the stream reception to the received stream outputin the sink device can be notified to the source device using thecontrol signal, the packet header, or a particular region of the packetpayload. The particular region of the packet payload can be designatedin the WFD capability negotiation 430 of FIG. 4 between the sourcedevice and the sink device.

FIG. 9 is a timing diagram illustrating exemplary timing for the variousoperations that occur in the method of FIG. 5A. For simplicity ofexplanation, an example of streaming a single video frame of a videostream is presented. At an initial time to, it is assumed that thesource device receives an encoded video frame from an external device orinternal component such as a camera, or retrieves the video frame fromdevice memory. (Herein, “receiving” such video or audio stream meanseither receiving it from an external source or from an internalcomponent or retrieving it from memory such as a video library.) At thistime, the source device temporarily stores the frame in RAM. Then, thesource device begins to decode the video frame at a time t₁ and displaysthe video frame at t₂. Thus a source device decoding delay T_(D)=timet₂−t₁ is computed by the source device using, e.g., one of thetechniques described above.

At time t₂ (after delaying some time), the source device begins totransmit the encoded video frame (and optionally, a corresponding audioframe) to the sink device. At this time t₁, the source device is capableof displaying the decoded frame on its screen, i.e., reproducing(outputting) the video frame. An analogous process can be performed forthe audio stream.

The operation of the source device for delaying the encoded video streamby the certain time from the frame time information of the encoded videostream and transmitting to the sink device can include packetizing thePlease confirm or correct.) The packetization can be MPEG-2-TS containerformat and encapsulate with a Real Time Protocol (RTP)/UniversalDatagram Protocol (UDP)/Internet Protocol (IP) header prior to thepacketization according to Institute of Electrical and ElectronicsEngineers (IEEE) 802.11 standard.

With continuing reference to FIG. 5A, the operation of the source devicefor checking the encoding format of the encoded video stream in step 510can include checking the encoding format of the encoded audio stream.That is, the source device checks whether the encoding format of theencoded audio stream, to be decoded and played at the source devicespeaker, is the decodable format of the sink device. The delaying andtransmitting to the sink device the encoded video stream by the certaintime from the frame time information of the encoded video stream in step520 can be done in conjunction with a process of decoding the frame ofthe encoded audio frame parsed and stored by the source device,outputting it through the speaker of the source device, delaying theencoded audio stream by the certain time from the frame time informationof the encoded audio stream information, and transmitting it to the sinkdevice. The processing of the audio stream can be analogous to that ofthe aforementioned video stream.

For handling the extracted audio stream, the method can alternativelyinclude an operation of the source device decoding and outputting(playing) the audio stream, and re-encoding and transmitting the outputaudio stream to the sink device (discussed earlier in connection with“differential processing”) Instruction sets for the respectiveoperations of FIG. 5A can be included in the CODEC module 215 of thememory 210 of FIG. 2. In this case, the CODEC module 215 stored to thememory 210 can be executed by one or more processors. The instructionsets can alternatively be stored in the application module 217.

FIG. 5B is a diagram of a general apparatus which can carry out methodof the flowchart of FIG. 5A, for establishing the WFD communicationsession for transmitting the video stream from the source device to thesink device according to an exemplary embodiment of the presentinvention. A means 530 of the source device for checking the encodingformat of the encoded video stream determines whether the encodingformat of the encoded video stream which is decoded by the source deviceand output through the output device of the source device, is thedecodable format of the sink device. A means 540 for outputting thedecoded video stream and transmitting the encoded video stream decodesthe frame of the encoded video stream, outputs the decoded frame throughthe output device of the source device, and transmits the encoded videostream by delaying it by the certain time from the frame timeinformation of the encoded video stream.

In addition, the apparatus can further include a means (not shown) forseparating the audio stream and the video stream by parsing the encodedstream which is the interleaved video stream and audio stream, a means(not shown) for receiving the encoding format of the video streamdecodable by the sink device from the sink device, a means (not shown)for sending the request inquiring about whether the encoding format ofthe encoded video stream is decodable by the sink device, and receivingthe response of the request from the sink device, and/or a means (notshown) for packetizing the encoded video stream according to thetransport stream standard.

In addition, the apparatus can include a means (not shown) of the sourcedevice for decoding and outputting the encoded audio stream, andencoding and transmitting the output audio stream to the sink device.

Such means can be implemented using separate or single hardware.

FIG. 6A is a flowchart of WFD communication session establishment fortransmitting the video stream from the source device to the sink deviceaccording to another exemplary embodiment of the present invention. Inan operation 610, the source device checks the encoding format of anencoded video stream to be displayed at the source device, anddetermines whether the encoding format is a decodable format of the sinkdevice. In an operation 620, the source device transmits the encodedvideo stream including delay information, and decodes and displays thevideo frame of the encoded video stream. In other words, the sourcedevice transmits the encoded video stream to the sink device whereinframe of the encoded video stream includes the delay information of thecertain time for delaying and outputting the encoded video stream by thecertain time from the frame time information of the encoded videostream. The delay information can be carried by the packet header or acertain region of the packet payload. A particular region of the packetpayload can be designated between the source device and the sink devicein the WFD capability negotiation 430 of FIG. 4. Also, a particularcontrol signal can carry the delay information.

Before the source device checks the encoding format of the encoded videostream in step 610, the encoding format of the video stream decodable bythe sink device can be received from the sink device. This operation canrefer to the corresponding explanations in FIG. 5A.

Before the source device checks the encoding format of the encoded videostream in step 610, the request inquiring about whether the encodingformat of the encoded video stream, which is decoded by the sourcedevice and output through the output device of the source device, is thedecodable format of the sink device can be sent from the source deviceto the sink device, and the response of the request can be received fromthe sink device. This operation can refer to the correspondingexplanations in FIG. 5A.

When the decoded video stream is output and the encoded video streamincluding the delay information is transmitted in step 620, before thesource electronic device decodes the frame of the encoded video stream,the audio stream and the video stream can be separated and stored byparsing the encoded stream of the interleaved video stream and audiostream (which can be referred to the video), which is not illustrated inthe drawing. This operation can refer to the corresponding explanationsin FIG. 5A.

FIG. 10 is a timing diagram illustrating exemplary timing of the variousoperations of the method of FIG. 6A. An example of streaming a singlevideo frame of a video stream is presented. At an initial time to, it isassumed that an encoded video frame is received or retrieved by thesource device. Then, att₁, the source device starts to decode the videostream and transmit the video stream to the sink device. Here, frame ofthe encoded video stream transmitted from the source device to the sinkdevice includes the delay information of the certain time for delayingand outputting the encoded video stream by the certain time from theframe time information of the encoded video stream. The delayinformation may be calculated according to the methods described inconnection with the FIG. 5A and FIG. 9.

With continued reference to FIG. 6A, before the source device transmitsto the sink device by delaying by the certain time from the frame timeinformation of the encoded video stream, the encoded video stream can bepacketized according to the transport stream standard. This operationcan refer to the corresponding explanations in FIG. 5A.

In addition, the operation 610 of the source device for checking theencoding format of the encoded video stream can include checking theencoding format of the encoded audio stream. That is, this operationdetermines whether the encoding format of the encoded audio stream whichis decoded by the source device and played through the speaker of thesource device, is the decodable format of the sink device. The operation620 for outputting the decoded video stream and transmitting the encodedvideo stream including the delay information, can decode and output theframe of the encoded audio stream stored by the source device by parsingthe encoded stream, through the output device of the source electronicdevice, and transmit the encoded audio stream to the sink device byincluding the delay information instructing to delay and output theencoded audio stream by the certain time from the frame time informationof the encoded audio stream. The explanations on the audio stream arebased on the video stream as mentioned above.

As an alternative to handling the extracted audio stream, as mentionedearlier in reference to differential processing, the source device candecode and play the audio stream, re-encode the output audio stream, andtransmit the re-encoded audio stream to the sink device.

Instruction sets for the respective operations of FIG. 6A can beincluded in the CODEC module 215 of the memory 210 of FIG. 2. In thiscase, the CODEC module 215 stored to the memory 210 can be executed byone or more processors.

FIG. 6B depicts an apparatus, which corresponds to the method of theflowchart of FIG. 6A, for establishing the WFD communication session fortransmitting the video stream from the source device to the sink deviceaccording to another exemplary embodiment of the present invention. Ameans 630 of the source device for checking the encoding format of theencoded video stream determines whether the encoding format of theencoded video stream which is decoded by the source device and outputthrough the output device of the source device, is the decodable formatof the sink device. A means 640 for outputting the decoded video streamand transmitting the encoded video stream including the delayinformation decodes and outputs the frame of the encoded video stream,through the output device of the source device, and transmits the videostream to the sink device by including the delay information instructingto delay and output the encoded video stream by the certain time fromthe frame time information of the encoded video stream.

In addition, the apparatus can further include a means (not shown) forseparating the audio stream and the video stream by parsing the encodedstream which is the interleaved video stream and audio stream, a means(not shown) for receiving the encoding format of the video streamdecodable by the sink device from the sink device, a means (not shown)for sending the request inquiring about whether the encoding format ofthe encoded video stream is decodable by the sink device, and receivingthe response of the request from the sink device, and/or a means (notshown) for packetizing the encoded video stream according to thetransport stream standard.

In addition, the apparatus can include a means (not shown) of the sourcedevice for decoding and outputting the encoded audio stream, andencoding and transmitting the output audio stream to the sink device.

Such means can be implemented using separate or single hardware.

FIG. 7A is a flowchart of the sink device, which receives the videostream from the source device, for receiving and outputting the videostream of the sink device according to another exemplary embodiment ofthe present invention. When confirming the output delay information ofthe received encoded video stream in step 710, the sink device confirmsthe certain time to delay for decoding and outputting the encoded videostream, from the packet (frame) of the encoded video stream receivedfrom the source electronic device. Next, when decoding and outputtingthe encoded video stream through the output device of the sink device bydelaying according to the delay information in step 720, the sink devicedelays and outputs the time for decoding and outputting the receivedencoded video stream according to the confirmed delay information.

The method for delaying and outputting can use a method for delaying thedecoding timing of the encoded video stream by a certain timecorresponding to the delay information.

Alternatively, the method can delay, after decoding the encoded videostream, the output timing of the output device by a certain timecorresponding to the delay information.

In addition, the sink device can receive encoded audio from the sourcedevice, decode and output the encoded audio stream as described earlier.

FIG. 7B illustrates an apparatus, corresponding to the flowchart of FIG.7A, for receiving and outputting the video stream of the sink devicewhich receives the video stream from the source device according toanother exemplary embodiment of the present invention. The sink deviceincludes a means 730 for confirming the output delay information of thereceived encoded video stream, and a means 740 for decoding andoutputting the encoded video stream through the output device bydelaying according to the delay information. The means 730 forconfirming the output delay information of the received encoded videostream confirms the certain time to delay for decoding and outputtingthe encoded video stream, from the packet (frame) of the encoded videostream received from the source device. The means 740 for decoding andoutputting the encoded video stream through the output device bydelaying according to the delay information delays and outputs the timefor decoding and outputting the received encoded video stream accordingto the confirmed delay information.

The apparatus can further include a means (not shown) for outputting thedecoded video stream by delaying the decoding timing of the encodedvideo stream by a certain time corresponding to the delay information,or a means (not shown) for delaying the output timing of the outputdevice, after decoding the encoded video stream, by a certain timecorresponding to the delay information.

In addition, the sink device can receive an encoded audio stream, anddecode and output the encoded audio stream as described above.

Such means can be implemented using separate or single hardware.

FIG. 8 is a flowchart of WFD communication session establishment fortransmitting the video stream from the source device to the sink deviceaccording to another exemplary embodiment of the present invention.

The method determines whether the encoded video stream of the sourcedevice is in the video stream format decodable by the sink device instep 810. That is, the method determines whether the encoding format ofthe encoded video stream which is decoded by the source device andoutput through the output device of the source device, is the decodableformat of the sink device. For doing so, the operating (not shown) forreceiving the encoding format (i.e., encoding format information) of thevideo stream decodable by the sink device, from the sink device can bepreceded. The sink device may decode a plurality of encoding formats ofthe video stream. The encoding format information of the video streamdecodable by the sink device can be received in, for example, the WFDcapability negotiation 430 of FIG. 4. That is, when determining thesequence of the message exchange between the source device and the sinkdevice and determining the parameters of the transmitted video/audiostreams, the encoding format information of the decodable video streamcan be received from the sink device. The sink device can transmit theencoding format information of the video stream using the controlsignal, the frame header (the packet header in the packet transmission),or the particular location of the frame payload (the packet payload ofthe packet transmission).

An operation (not shown) for sending from the source device to the sinkdevice the request inquiring about whether the encoding format of theencoded video stream, which is decoded by the source device and outputthrough the output device of the source device, is the decodable formatof the sink device, and receiving the response of the request from thesink device can be preceded. This operation can be performed in, forexample, the WFD capability negotiation 430 of FIG. 4.

When the encoded video stream of the source device is in the decodableformat of the sink device, the method confirms the frame (packet) timeinformation of the encoded video stream stored to the source device, andconfirms the frame (packet) time information of the video stream decodedto output the encoded video stream in step 820. (The video stream isconcurrently decoded at the source device and output on the screen ofthe source device at step 820.)

The encoded video stream is obtained by separating and storing theencoded audio stream and the encoded video stream by parsing the encodedstream (the video) which is the interleaved encoded video stream andaudio stream.

The encoded stream can be, but not limited to, downloaded from otherdevice and stored to the source device, obtained from the component suchas camera subsystem of the source device, or streamed over the Internet.The parsing can be realized using hardware or software. Using thesoftware, the instruction set for the parsing function can be providedas a separate parsing module or as part of the CODEC module and thenstored to the memory 210 of FIG. 2. The parsed video stream and audiostream are stored to the source electronic device. Their storagelocation can be part of the memory interface, part of the peripheralinterface, or a separate buffer memory (RAM).

The frame time information of the encoded video stream stored to thesource device can be obtained from the time information of the wholepacket and/or the time information per frame, in the packet header(frame header) or the packet payload (frame payload) of the parsedencoded video stream. The frame time information corresponds to the PPSand the SPS in the MPEG format. The corresponding frame time informationindicates the output order of the corresponding frame. The frame timeinformation of the decoded video stream to output the encoded videostream can be acquired by monitoring the output frame of the decodedvideo stream. Hence, the difference value can be determined by comparingthe frame time information of the encoded video stream and the frametime information of the decoded and output video stream.

The difference value can be produced by subtracting the time taken fromthe transmission of the source device to the reception of the sinkdevice, from the difference value of the frame time information of theencoded video stream stored to the first storage device and the frametime information of the video stream decoded to output the encoded videostream. The time taken from the transmission of the source device to thereception of the sink device can be ½ of the RTT value between thesource device and the sink device.

The difference value may be produced by subtracting the time taken bythe transmission of the source device and the sink device and the timetaken for the sink device to receive, decode, and output the encodedvideo stream through its output device, from the difference value of theframe time information of the encoded video stream stored to the firststorage device and the frame time information of the video streamdecoded to output the encoded video stream. The time taken for the sinkdevice to receive, decode, and output the encoded video stream throughits output device can be measured by the sink device. This particulartime value can be notified to the source device using the controlsignal, the packet header, or the particular region of the packetpayload. The particular region of the packet payload can be designatedduring the WFD capability negotiation 430 of FIG. 4 between the sourcedevice and the sink device.

Next, in step 830, the method snoops the encoded video stream byapplying the difference value of the frame (packet) time information ofthe encoded video stream and the frame (packet) time information of thedecoded video stream to output. The difference value can use the variousvalues as explained earlier. The snooping of the encoded video streamincludes delaying the encoded video stream by the difference value instep 830.

In step 840, the encoded video stream is packetized according to thetransport stream standard. The packetization can employ, for example,the MPEG-2-TS container format and encapsulate with the RTP/UDP/IPheader prior to the packetization according to the IEEE 802.11 standard.

In step 850, the source device transmits the encoded video stream to thesink device.

In addition, when the method determines whether the encoded video streamof the source device is in the video stream format decodable by the sinkdevice in step 810, the encoding format of the encoded audio stream canbe checked. That is, the method determines whether the encoding formatof the encoded audio stream which is decoded by the source device andoutput through the output device of the source device, is the decodableformat of the sink device. The subsequent processing on the audio streamcan be explained based on the descriptions of the video stream as statedabove.

The source device can further decode and output the encoded audiostream, and encode and transmit the output audio stream to the sinkdevice, which is not depicted in the drawing.

If at step 810, it is determined that the encoded video stream format isnot compatible with any decoding format at the sink device, the methodproceeds to step 860. Here, the video stream is decoded and output atthe source device, and the decoded video is then re-encoded in a formatcompatible with a decoding format of the sink device at step 870 in aconventional manner. The re-encoded video stream is then packetized andtransmitted at steps 840 and 850.

The instruction sets for the respective operations of FIG. 8 can beincluded in the CODEC module 215 of the memory 210 of FIG. 2. In thiscase, the CODEC module 215 stored to the memory 210 can be executed byone or more processors.

The exemplary embodiments and various functional operations of thepresent invention described herein can be implemented in computersoftware, firmware, hardware, or in combinations of one or more of themincluding the structures disclosed in this specification and theirstructural equivalents. The exemplary embodiments of the presentinvention can be implemented as one or more computer program products,that is, one or more data processors, or one or more modules of computerprogram instructions encoded on a computer-readable medium to controlthe devices.

The computer-readable medium may be a machine-readable storage medium, amachine-readable storage substrate, a memory device, a materialaffecting a machine-readable propagated stream, or a combination of oneor more of these. The term ‘data processor’ encompasses every device,apparatus, and machine including, for example, a programmable processor,a computer, a multiple processors, or a computer, for processing data.The device can be added to the hardware and include program code forcreating an execution environment of a corresponding computer program,for example, code for constituting processor firmware, a protocol stack,a database management system, an operating system, or a combination ofone or more of these.

The propagated stream is an artificially generated stream, for example,a machine-generated electrical, optical, or electromagnetic stream whichis generated to encode information for transmission to a suitablereceiver. A computer program (which is also referred to as a program,software, software application, script, or code) can be written in anyform of a programming language including compiled or interpretedlanguages, and can be deployed in any form including a stand-aloneprogram or a module, component, subroutine, or other unit suitable foruse in a computing environment. A computer program may not need tocorrespond to a file of a file system. A program can be stored in aportion of a file which holds other programs or data (e.g., one or morescripts stored in a markup language document), in a single filededicated to the corresponding program, or in multiple coordinated files(e.g., one or more modules, sub-programs, or part of code).

A computer program can be deployed to be executed on one computer or onmultiple computers which are located at one site or distributed acrossmultiple sites and interconnected by a communication network. Theprocesses and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating the output. The processes and logic flows can also beperformed by, and the apparatus can also be implemented as, specialpurpose logic circuitry, for example, a Field Programmable Gate Array(FPGA) or an Application Specific Integrated Circuit (ASIC).

Although this specification contains many specific details, these shouldnot be construed as limitations on the scope of any inventions or ofwhat may be claimed, but rather as descriptions of features specific toparticular embodiments of particular inventions. Certain features whichare described in this specification in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features described in the context of asingle embodiment can also be implemented in multiple embodimentsseparately or in any suitable subcombination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination may be directed to a subcombination or variation ofa subcombination. Similarly, although operations are depicted in thedrawings in a particular order, this should not be understood asrequiring that such operations be performed in the particular ordershown or in sequential order, or that all illustrated operations beperformed, to achieve desirable results. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system components in the embodiments describedabove should not be understood as requiring such separation in allembodiments, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products.

While the invention has been shown and described with reference tocertain exemplary embodiments thereof, it will be understood by thoseskilled in the art that various changes in form and details may be madetherein without departing from the spirit and scope of the invention asdefined by the appended claims and their equivalents.

What is claimed is:
 1. A method operable in a first electronic device, comprising: determining whether an encoding format of an encoded video stream is a format decodable by a second electronic device; and decoding and outputting a frame of the encoded video stream through an output device of the first electronic device, and transmitting the encoded video stream to the second electronic device, wherein the transmitting the encoded video stream to the second electronic device is transmitting the encoded video stream without re-encoding it after delaying by a determined time from frame time information of the encoded video stream.
 2. The method of claim 1, wherein the determined delay time is based at least in part on a determined decoding time of the video stream at the first electronic device.
 3. The method of claim 1, wherein the determined delay time is based at least in part on a determined decoding time of the video stream at the first electronic device and a link delay in transmitting the encoded video stream to the second electronic device.
 4. The method of claim 1, wherein the determined time delay is based at least in part on a determined decoding time of the video stream at the first electronic device, a link delay in transmitting the encoded video stream to the second electronic device, and a decoding time in decoding the encoded video stream at the second electronic device.
 5. The method of claim 1, further comprising, before determining whether the encoding format is a decodable format: receiving an encoding format of a video stream decodable by the second electronic device, from the second electronic device.
 6. The method of claim 1, further comprising, before determining whether the encoding format is a decodable format: sending from the first electronic device to the second electronic device a request inquiring about whether the encoding format of the encoded video stream is a decodable format of the second electronic device; and receiving, at the first electronic device, a response of the request from the second electronic device.
 7. The method of claim 2, wherein the determined decoding time is obtained based on frame time information of the encoded video stream obtained from a header of the encoded video stream stored after parsing an encoded A/V stream to obtain the encoded video stream and an encoded audio stream.
 8. The method of claim 2, wherein the determined decoding time is obtained based on frame time information of the decoded video stream obtained by monitoring an output frame of the decoded video stream.
 9. The method of claim 1, wherein the determined delay time is based at least in part on a time determined by comparing the frame time information of the encoded video stream, which is obtained from the header of the encoded video stream stored after parsing the encoded video stream, and frame time information of the decoded video stream, which is obtained by monitoring an output frame of the decoded and displayed video stream.
 10. The method of claim 1, wherein the encoded video stream is transmitted to the second electronic device by packetizing the encoded video stream according to a transport stream standard.
 11. The method of claim 1, wherein the encoded video stream is part of an encoded audio/video (A/V) stream, and the method further comprising: determining whether an encoding format of an audio stream of the A/V stream is a format decodable by the second electronic device, if so, transmitting the encoded audio stream without re-encoding it to the second device.
 12. The method of claim 1, further comprising: decoding and outputting an encoded audio stream in the first electronic device, and re-encoding and transmitting the output audio stream to the second electronic device.
 13. A first electronic device comprising: one or more processors; a memory; and one or more modules stored to the memory to be executed by the one or more processors, wherein the one or more modules stores instructions for determining whether an encoding format of an encoded video stream is a format decodable by a second electronic device and transmitting the encoded video stream to the second electronic device wherein the transmitting the encoded video stream to the second electronic device is transmitting the encoded video stream without re-encoding it after delaying by a determined time from frame time information of the encoded video stream.
 14. The first electronic device of claim 13, wherein the determined delay time is based at least in part on a determined decoding time of the video stream at the first electronic device.
 15. The first electronic device of claim 13, wherein the determined delay time is based at least in part on a determined decoding time of the video stream at the first electronic device and a link delay in transmitting the encoded video stream to the second electronic device.
 16. The first electronic device of claim 13, wherein the determined time delay is based at least in part on a determined decoding time of the video stream at the first electronic device, a link delay in transmitting the encoded video stream to the second electronic device, and a decoding time in decoding the encoded video stream at the second electronic device.
 17. The first electronic device of claim 13, wherein the one or more modules further stores an instruction for discerning reception of an encoding format of a video stream decodable by the second electronic device, from the second electronic device.
 18. The first electronic device of claim 13, wherein the one or more modules further stores instructions for sending to the second electronic device a request inquiring about whether the encoding format of the encoded video stream is a decodable format of the second electronic device, and for discerning reception of a response of the request from the second electronic device.
 19. The first electronic device of claim 14, wherein the determined decoding time is based on frame time information of the encoded video stream obtained from a header of the encoded video stream stored after parsing the encoded video stream.
 20. The first electronic device of claim 14, wherein the determined decoding time is based on frame time information of the decoded video stream obtained by monitoring an output frame of the decoded video stream.
 21. The first electronic device of claim 13, wherein the determined delay time is based at least in part on a time determined by comparing the frame time information of the encoded video stream, which is obtained from the header of the encoded video stream stored after parsing the encoded video stream, and frame time information of the decoded video stream, which is obtained by monitoring an output frame of the decoded and displayed video stream.
 22. The first electronic device of claim 13, wherein the one or more modules further stores an instruction for packetizing the encoded video stream according to a transport stream standard.
 23. The first electronic device of claim 13, wherein the encoded video stream is part of an audio/video (A/V) stream, and the instructions further comprise instructions for: determining whether an encoding format of an audio stream of the A/V stream is a format decodable by the second electronic device, if so, transmitting the encoded audio stream without re-encoding it to the second device.
 24. The first electronic device of claim 13, comprising a touch screen including a display that displays the decoded video stream.
 25. A computer-readable storage medium storing one or more programs comprising instructions which, when executed by an electronic device, cause the device to execute the method according to claim
 1. 